- ❌ Array jobs monitored with malformed, multi-token SLURM job IDs.
- ⚠️ Frontier submit wrapper unreliable for scripts using --array.
- ⚠️ Downstream monitoring scripts may misreport job status.

1. In the repo root, create a SLURM batch script `array_job.sh` that defines an array job,
e.g.:

   - `#SBATCH --array=0-3`

   This script will later be passed as `$1` to `.github/workflows/frontier/submit.sh`.

2. On a SLURM system, run the wrapper:

   - `.github/workflows/frontier/submit.sh array_job.sh cpu`

   This executes the logic in `.github/workflows/frontier/submit.sh` including the
   `sbatch` submission block and captures `sbatch` output into `submit_output`.

3. SLURM prints a message like `Submitted batch job 123456_0` (typical for array jobs) to
stdout. The script at `.github/workflows/frontier/submit.sh:31-58` captures this into
`submit_output`, then the block at lines 61-66 executes:

   - `job_id=$(echo "$submit_output" | grep -oE '[0-9]+')`

   `grep -oE '[0-9]+'` returns two lines: `123456` and `0`, so `job_id` contains a
   newline-separated list of two numeric tokens.

4. The monitoring code at `.github/workflows/frontier/submit.sh:71-72` runs:

   - `bash "$SCRIPT_DIR/../../scripts/monitor_slurm_job.sh" "$job_id" "$output_file"`

   Because `"$job_id"` contains a newline, `monitor_slurm_job.sh` receives an invalid job
   identifier (either as a value with embedded newline or as multiple numeric tokens,
   depending on its parsing). This causes incorrect monitoring behavior for the submitted
   job array (e.g., failing SLURM commands like `squeue -j` or tracking the wrong job),
   demonstrating that extracting all numeric substrings from `submit_output` produces an
   invalid job ID when multiple numbers are present.

This is a comment left during a code review.

**Path:** .github/workflows/frontier/submit.sh
**Line:** 61:61
**Comment:**
	*Logic Error: The job ID extraction uses `grep -oE '[0-9]+'` directly on the `sbatch` output, which returns every numeric substring on separate lines; if the output ever contains more than one number, `job_id` will contain multiple IDs separated by newlines, causing the monitoring script to receive an invalid job identifier and fail to track the job correctly. Limiting the extraction to the first numeric match avoids this and ensures `job_id` is a single valid token.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.

- ❌ monitor_slurm_job.sh may track wrong SLURM job.
- ⚠️ Frontier CI runs may be misreported as successful.

1. On a Frontier login node, create a simple SLURM script `run.sh` in the repo root that
just echoes a message and exits 0.

2. Run `.github/workflows/frontier_amd/submit.sh run.sh cpu` so that
`.github/workflows/frontier_amd/submit.sh:1-72` executes and calls `sbatch <<EOT` to
submit the job.

3. On clusters where `sbatch` prepends informational or warning lines containing digits
(e.g., maintenance date or policy message) before the standard `Submitted batch job NNNNN`
line, those lines become part of `submit_output` at
`.github/workflows/frontier_amd/submit.sh:34-59`.

4. At `.github/workflows/frontier_amd/submit.sh:61`, `job_id=$(echo "$submit_output" |
grep -oE '[0-9]+')` captures *all* numeric tokens (one per line), so `job_id` becomes a
multi-line string like `2025\n02\n10\n12345`; when passed to
`.github/scripts/monitor_slurm_job.sh` at lines 71-72 as the first argument, the
monitoring script is given an invalid or wrong job ID and will monitor the wrong job or
fail to monitor at all.

This is a comment left during a code review.

**Path:** .github/workflows/frontier_amd/submit.sh
**Line:** 61:61
**Comment:**
	*Logic Error: The job ID extraction uses a bare `grep -oE '[0-9]+'` on the entire sbatch output, so any digits from additional messages (e.g., debug logs or some error texts that still return exit 0) can be misinterpreted as the job ID, causing the monitor to track the wrong job or think submission succeeded when it did not. Restrict the parsing to the standard "Submitted batch job NNNNN" line and extract the numeric field from that line only.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.

- ❌ Failed sbatch treated as successful job submission.
- ⚠️ monitor_slurm_job.sh tracks nonexistent or wrong SLURM job.
- ⚠️ Users misled about status of cluster submissions.

1. On a SLURM cluster, in the repo root, create a job script `bad_job.sh` whose SBATCH
options cause `sbatch` to reject it but emit an error containing digits (for example an
invalid time limit), ensuring `.github/workflows/phoenix/submit.sh` can read it at lines
12–15.

2. Run the submission helper: `.github/workflows/phoenix/submit.sh bad_job.sh cpu none`;
this calls `sbatch` via the here‑document started at line 40 and ended at line 66,
capturing all `sbatch` stdout/stderr into `submit_output`.

3. Because `sbatch` rejected the job, its output does not contain the normal `Submitted
batch job <id>` line, but does contain numeric tokens in the error message; at line 69,
`grep -oE '[0-9]+'` extracts all digit sequences from that error so `job_id` becomes a
concatenation of those numbers, and the non‑empty check at lines 69–73 passes as if
submission succeeded.

4. The script then prints `Submitted batch job $job_id` (line 76) and invokes
`scripts/monitor_slurm_job.sh` with this bogus job ID at lines 79–80, causing the
monitoring script to watch a non‑existent or wrong job and never report the actual
submission failure.

This is a comment left during a code review.

**Path:** .github/workflows/phoenix/submit.sh
**Line:** 69:69
**Comment:**
	*Logic Error: The job ID is parsed by grepping for any number in the sbatch output, so if sbatch prints additional numeric tokens (for example in warnings or extra messages), the script may capture multiple numbers or the wrong one, causing the monitoring script to track an invalid job or fail; instead, parse the specific "Submitted batch job" line and extract only the job ID field.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.